Eecient Real-time Index Updates in Text Retrieval Systems

As information retrieval (IR) systems emerge as the mainstream information nding tool within commercial enterprises due to the enormous popularity of World Wide Web (WWW) technology in the intranet environments, the ability to incorporate new and/or updated documents into the database in real time becomes an essential requirement. However, conventional IR systems are optimized for read queries by employing aggressive indexing and/or query result caching. The cost of maintaining the consistency between the indexes and the underlying database is expensive but assumed to be tolerable since update processing is performed in a batch fashion. This paper describes the design and implementation of a real-time index/cache consistency maintenance technique in Codir, a text retrieval system that features an integrated compression and and indexing scheme. The main idea behind Codir's real-time index consistency maintenance technique is to build transient index for new document updates, and to process read queries using both permanent and transient index. To minimize the performance overhead associated with document database updates, Codir integrates transient index with permanent index lazily either by piggybacking the integration task with read query processing or by periodic batch processing. Finally, unlike previous IR systems, Codir supports document deletion, based on a lazy invalidation approach.